Increasing the resolution of an image is commonly known as upscaling and is becoming of increasing interest. For example, due to the advent of various High Definition (HD) television standards, there is a desire for suitable methods and algorithms for generating HD images from Standard Definition (SD) images. For example, offline processing has been used to generate HD video sequences from original SD video sequences and television and DVD players etc have even been developed that can dynamically upscale SD video content to HD video content in real time.
Such upscaling typically uses interpolation to generate new pixel values using e.g. bilinear or bicubic interpolation, or polyphase scaling. However, although such methods increase the number of pixels in the image, they do not introduce new high frequency detail in the image and accordingly the upscaled images appear blurred. For conversion of standard definition (SD) video to high definition (HD) an upscaling factor of approximately 200% is required and for this and higher scale factors, the blurring tends to be clearly visible and undesirable.
In order to mitigate or compensate such perceived blurring, advanced non-linear techniques are often used in resolution enhancement to produce an image with a more detailed and sharp impression. Such methods include for example data dependent interpolation filters optimized using training and structure classification as described in T. Kondo, and K. Kawaguchi, “Adaptive dynamic range encoding method and apparatus”, U.S. Pat. No. 5,444,487, August 1995. and T. Kondo et al., “Method and apparatus for adaptive filter tap selection according to a class”, U.S. Pat. No. 6,192,161, February 2001. edge directed interpolation techniques as described in “New edge-directed interpolation.” by Li and Orchard, IEEE Transactions on Image Processing 10: 1521-1527 2001. An overview and evaluation of many of these techniques can be found in “Image super-resolution survey.”, by van Ouwerkerk; Image and Vision Computing 24(10): 1039-1052, 2006 and “Making the Best of Legacy Video on Modern Displays” Journal of the Society for Information Display—January 2007 Volume 15, Issue 1, pp. 49-60 by de M. Zhao, M. Bosma, and G. de Haan.
A specific example of a method for resolution enhancement used in televisions is known as Luminance Transient Improvement (LTI) with peaking LTI is a technique that improves the sharpness of edges by increasing the transient without creating overshoots thereby introducing new high-frequency content in the signal. Peaking is aimed at boosting the high-frequency components already present in the signal in order to give the images a sharper impression. A description of LTI can be found in J. Tegenbosch, P. Hofman and M. Bosma, “Improving nonlinear up-scaling by adapting to the local edge orientation”, Proceedings of the SPIE, Vol. 5308, pp. 1181-1190, January 2004 and U.S. Pat. No. 4,414,564 A.
However, typically these methods mainly focus on generating sharper edge transitions and as a consequence they tend to lack sufficient enhancement capabilities in densely detailed areas such as textures. This lack of texture-/detail-enhancement can become more visible for increasing scaling factors and in particular for scaling factors above 200%.
Methods for texture synthesis have been proposed in e.g. “Texture synthesis by fixed neighborhood searching” by Wei, L.-Y., 2002 and “Fast Texture Transfer” by Ashikhmin, M; Computer Graphics and Applications, IEEE Volume 23, Issue 4, July-August 2003 Page(s): 38-43. The first of these articles disclose an example of the basic example-based texture synthesis which is a technique aimed at expanding texture in an image area based on a small texture patch being used as an example. However, although such methods are useful for covering an area by a suitable texture they are not directed to upscaling or resolution enhancement and are directed towards expanding a texture to a wider area rather than increasing the resolution of an existing texture area. The second document discloses a technique known as texture transfer (or sometimes called constrained texture synthesis) where the synthesis is steered by an extra target image.
Another method for upscaling is known as example-based super-resolution and is described in “Example-Based Super-Resolution.”; Freeman, W. T.; Jones, T. R.; Pasztor, E. C., Computer Graphics and Applications, IEEE, vol. 22, no. 2, pp. 56-65, March/April 2002 and William T. Freeman and Thouis R. Jones, “One-pass super-resolution images” U.S. Pat. No. 6,766,067 Jul. 20, 2004.
Example-based super-resolution uses a database with sets of pre-stored images where each set comprises a low-resolution and a high-resolution version of the same scene. The input image to be upscaled is processed patch-by-patch and for each patch a best matching example is found in the database. The match is determined by comparing the low resolution content of the patch and the corresponding low resolution patches stored in the database for the example images. Furthermore, already synthesized pixel values of the upscaled image may be compared to pixels stored in the database as high resolution image examples corresponding to the low resolution match. This is achieved by letting the synthesized patches have a small overlap, which in effect imposes better spatial consistency. The high frequency components of the stored high resolution image for the best match are then used to synthesize the high-resolution output thereby generating an improved upscaled image. Specifically, pixel values may be copied from the stored high resolution image of the best matching example.
However, although this approach may provide good results for some images, it also tends to have some disadvantages. In particular, the generated high resolution images may not have optimal quality. For example, because the process is performed on a patch-by-patch basis, the spatial consistency in the output is not guaranteed. Furthermore, the example images may not accurately correspond to the image being processed thereby introducing inconsistencies. Also, in order to achieve a sufficiently high quality for many different input images, it is necessary to have a large number of example images stored thereby increasing resource requirements, complexity and cost. For example, memory requirements and search size and time characteristics may be high.
Hence, an improved image resolution enhancement would be advantageous and in particular a system allowing increased flexibility, reduced complexity, improved image quality, reduced resource consumption and/or improved performance would be advantageous.